The present invention relates to recording, sharing, and communicating information associated with an imager to define the system behavior. It includes how to create wide-angle image or video encoded with information by wide-angle imagers and how to automatically decode and apply the information and image processing or display. The information is ultimately used to define a specific system behavior or user experience based on the parameters.
Using image or video, a user can record, share, or communicate information. In addition to the image or video content, there is also a lot of information associated with the content. This information is in the form of imager parameters, such as, but in no way to limited to, wide-angle device parameters, environment parameters, image processing, enhancement or display parameters, system behavior, user experience, or any other information to be recorded, shared, or communicated.
An existing well-known method to transfer some important parameters from an imager to an end user is the use of metadata and the like. Metadata is text information usually written in the header of a digital image file. This header is a part of the file before or after the actual image data particular to each file format. However, metadata is subject to loss of information when the image or video file is converted to another format or when it is transferred through different media, such as the Internet. It is also impossible to transfer metadata through a printed or scanned image. Thus, the use of metadata and the like is not a robust method to transfer information associated with wide-angle images or videos.
One kind of important information associated with wide-angle image or video are the imager parameters. Wide-angle imagers produce two-dimensional (2D) images from a real-world environment by remapping the object to the image, creating distortion or discontinuity in the images produced. No matter the kind of wide-angle imager used, from lenses to catadioptric systems and including imagers stitching some narrow-angle images together or imagers producing separated unstitched narrow-angle images, the final wide-angle image will always include some distortion or image remapping to present an environment in a 2D plane. The consequence of image remapping is that images are deformed in such a way that straight lines in the object environment appear curved, piecewise linear, or discontinuous in the wide-angle image depending on the type of imager.
To properly display a part of the image to a user or to an algorithm, a correction of the distortion is usually done as in U.S. Pat. No. RE44,087 E1 for linear distortion or U.S. Pat. Nos. 6,844,990 B2 and 6,865,028 B2 for custom symmetric and custom asymmetric distortion, respectively. To properly correct the distortion, the algorithm has to know the different parameters of the imager, including the exact distortion profile, the field of view, the position of the optical center of the image, and the spatial orientation of the wide-angle imager. However, with the above prior art, the distortion processing algorithm is either tied to a single wide-angle imager with the parameters pre-coded inside or requires manual selection of different parameters from an end user. The end user often doesn't have the exact knowledge of the parameters, making the distortion processing tedious and imperfect and a process to transfer the required parameters is required.
In the case of stereo image processing, the use of a bar-code added in the margin of an image file combining the two required images for stereo vision has been proposed in U.S. Pat. No. 8,358,329 B2 to store information from the imager, like the type of stereo image used and the distance between the viewpoints of both images. However, the transmitted data encoded inside the bar-code includes very limited parameters only for stereo images that are combined at the software level by an image synthesis unit.
Lens identification using bar-codes in cameras using analog films to capture images instead of digital sensors has also been proposed in U.S. Pat. Nos. 5,461,440 A and 6,707,998 B2. However, the static encoded data allows only for lens identification and cannot be used dynamically or to transmit changing imager information depending on the specific user application. Similarly, in other systems that use bar-codes or binary codes, the data encoded inside the bar-codes or binary codes are set by the camera manufacturer for a very specific purpose. The output being tied to a single use, it cannot change the output depending on the kinds of parameters that are present in the encoded data or depending on the specific user needs. Furthermore, these systems are static and cannot be used to dynamically encode or transmit changing imager parameters. Because of all these reasons, the encoded data cannot be used to define different system behaviors or user experiences depending on the specific information of the original scene. For optimal wide-angle image processing and to allow the defining of different system behaviors or user experiences depending on the parameters, the information could be added in real-time at the imager level to account for dynamically changing lens, camera, device or environmental parameters, such as the orientation, sub-region of the image transmitted, date, geographical location, or even dynamic distortion as proposed in U.S. Patent Publication No. 2012/0314065 A1.
In International Patent Publication No. WO 2007/110697 A3, a system was proposed to physically mark the optical system or optically add a marker in the image plane to allow identifying a wide-angle optical system. However, if a physical marker on the lens is used, this system still need a manual user external input of parameters at the imager such as the lens identification number or the orientation of the device and the like. When an optical marker is used to appear on the image plane, the system is static and is very limited since it cannot include dynamic encoding of the wide-angle imager parameters like the orientation or the sub-region of the image transmitted. It cannot transmit adapting imager information that depends on the specific user application and thus cannot be used to define different user experience or system behavior. Furthermore, another drawback is that a lens has to be designed from the beginning to include such markers and this method cannot be extended to already existing wide-angle imagers.
In addition to wide-angle device parameters, the present invention provides a scheme to record, share or communicate all kinds of lens, camera, device or environmental information associated with image or video. The scheme can be used in a much wider range of applications. The main innovation is then to be able to define different user experiences or system behaviors automatically by instructing what and how to output depending on the specific parameters. Furthermore, some information is dynamic, such as real-time camera orientation, real-time user experience, real-time system behavior and the dynamic information that cannot be recorded by the static way in the above patents.